Segmentation Based Interest Points and Evaluation of Unsupervised Image Segmentation Methods

This paper investigates segmentation based interest points for matching and recognition. We propose two simple methods for extracting features from the segmentation maps, which focus on the boundaries and centres of the gravity of the segments. In addition, this can be considered a novel approach for evaluating unsupervised image segmentation algorithms. Former evaluations aim at estimating segmentation quality by how well resulting segments adhere to the contours separating ground-truth foregrounds from backgrounds and therefore explicitly focus on particular objects of interest. In contrast, we propose to measure the robustness of segmentations by the repeatability of features extracted from segments on images related by various geometric and photometric transformations. Our evaluation provides a new insight into suitability of the segmentation methods for generating local features for image retrieval or recognition. Methodology. This study focuses on gauging performance of Efficient Graph-Based Image Segmentation (EGO), Mean Shift (MS), modified Watershed (WA) and modified Normalised Cuts (NC) in terms of their stability. It also compares them to state-of-the art MSER and Hessian [3] interest point detectors. Inspired by evaluation of affine region detectors [2, 3], we focused on two kinds of key-points locating potentially salient parts of segments. Ellipses inscribed in the segments are potentially repeatable features. Centre estimation and ellipse fitting can be performed on either contour coordinates or over the whole area. We found that area fitted ellipses are more repeatable as associated segments often suffer from partial spilling into noisy structures under both geometric or photometric changes. Corners located on region boundaries are salient features which may overcome the spilling problem. SUSAN detector [4] is very well tailored to detect corners and junctions on segment boundaries. Our implementation of SUSAN detector simply scans the boundaries of the segments with a 19x19 window and counts the number of pixels with the same label as the central point. Finally, non-minima are suppressed, and corners and junctions detected. Figure 1(bottom right) illustrates both types of features. To quantify performance, we exploited a set of well-known test images from [3]. Each image sequence consisted of 6 images with gradual distortions: bike/blur, boat/scale-rotation, car/illumination, graffiti/affine, house/JPEG compression, bark/zoom-rotation, tree/blur, wall/affine. We also employed two complementary measures based on the homography ground-truth. The region overlap from [3] was defined by a ratio of intersection to union of reference region Rr and projected region Rp: εo = 1− Rr∩Rp Rr∪Rp . This measure was used to evaluate centre based regions by the percentage of correspondences for which εo ≤ 0.3. For the boundary based points the correspondences were considered correct if the distance between the interest point and its nearest projected correspondence satisfied εn ≤ 4 pixels. The nearest neighbour (NN) repeatability measure was applied to quantify the accuracy of segment boundaries. The goal of adopting the overlap based repeatability [3] was to examine to what extent segments from a given segmentation are roughly preserved over a wide range of transformations. Figure 1 visualises the overlap (bottom third from left) and distance (bottom right) based correspondences. It is unclear how segmentations can be compared provided wide range of their tweaking parameters. Enforcing arbitrary number of segments does not guarantee appropriate scale of observation. To address this issue we adopted a simple ad hoc solution which uses EGO to generate three different control sets of segment maps at different scales of observation, namely: over-, well-, and under-segmented. The remaining segmentations were tweaked to fit to the control sets to their best abilities. In order to avoid damaging effect of exact fitting, we built histograms of sizes of segments for all tested methods and all images from the control sets. The segmentation parameters which produced the most similar histograms to the control set according to Chi2 distance were selected. Finally, we used three sets of parameters for each method. Figure 1: (Top) Segmentation maps of EGO, MS, NC, and WA on wellsegmented set (left to right). (Bottom) Segment (third from left) and boundary (rightmost) based features (yellow) in the reference image together with their correspondences (black) projected from another image.